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Abstract. We extend some results about shifted Schur functions to the general 
context of shifted Macdonald polynomials. We strengthen some theorems of F. Knop 
and S. Sahi and give two explicit formulas for these polynomials: a g-integral repre- 
sentation and a combinatorial formula. Our main tool is a (/-integral representation 
for ordinary Macdonald polynomial. We also discuss duality for shifted Macdonald 
polynomials and Jack degeneration of these polynomials. 



§1 Introduction 
The orthogonality of Schur functions 

(Sfj,, s\) = 5 uX , 

is the orthogonality relation for characters of the unitary group U(n). The orthog- 
onality of characters means that a character (as a function on the group) vanishes 
in all but one irreducible representation. 

There is a very remarkable basis in the center of the universal enveloping alge- 
bra U(qI(ti)), which is similar to the basis of characters in the algebra of central 
functions on U(n). Many properties of this basis can be found in [00,0k]. The 
elements of this basis are indexed by partitions (jl with at most n parts; they are 
denoted by and called quantum immanants. The element §y has degree and 
vanishes in as many irreducible representations, as possible, namely 

s* (A) = 0, unless /iQ 
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where s*(A) is the eigenvalue of 8^ in the representation with highest weight A. 
The function s*(A) is called the shifted Schur function for it is a polynomial in A 
with highest term s M (A). This polynomial is symmetric in variables 

Xi-i, i = l,2,..., 

and is also stable, which means it does not change if we add some zeroes to A. The 
number s*(A) has a remarkable combinatorial interpretation: up to some simple 
factors, it counts standard tableaux on X/ fx. 

There is the following formula for §> M in terms of the generators Eij of U(gl(n)). 
Fix any standard tableau T on fx. Let c T (i) denote the content of the i-th square 
in T. Then we have (see [Okl], and also [N,Ok2]) 

(1.1) §m = tr {{E - c T (l)) ® • • • <g> (E - c T (\fx\)) ■ P T ) , 
where E is the following matrix with entries in U(gl(n)) 

E = (Etj)^ e Mat(n) ® W(fll(ra)) , 
.Pr is the orthogonal projection on the Young basis vector indexed by T 

and we use the standard representation 

QS{\fx\) -> Mat(ra)®W . 

The formula (1.1) corresponds to the following combinatorial formula for the shifted 
Schur functions (see [00], §11, and also [Okl], §3.7). We call a tableau T on a 
diagram fx a reverse tableau if its entries strictly decrease down the columns and 
weakly decrease in the rows. Denote by T(s) the entry of T in the square s and by 
c(s) the content of the square s. Then we have 

(1.2) s* {x u x 2 , ■ ■ ■ ) = II ( Xt m ~ c ( s )) ' 

where T ranges over all reverse tableau on fx. 

The shifted Schur functions have numerous applications to the finite- and espe- 
cially infinite-dimensional representation theory. Their (q, t)-analogs, with which 
we deal in this paper, inherit most of their power. 

In the general (q, t)-situation the center of U(gl(n)) gets replaced by the com- 
mutative algebra generated by Macdonald (/-difference operators [M]. The eigen- 
functions of this commutative algebra are Macdonald polynomials P\(q,t), which 
replace Schur functions. The eigenvalue of a central element in the representation 
with highest weight A becomes the eigenvalue of a (/-difference operator on the Mac- 
donald polynomial P\(q,t). It is known (see, for example [EK]) that the algebra 
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generated by Macdonald operators can be naturally identified with the center of 
the (/-deformed U(gl(n)). 

The eigenvalue of a Macdonald operator on P\(q : t) is known to be a polynomial 
in q Xi which is symmetric in variables 

q K t~\ 

(q, £)-analog of the shifted Schur function should be a poly- 

degP*(x) = , 

variables 

Xi t , i — 1,2,..., 
and satisfies the following vanishing condition 

(1.3) P*(q x )=0, unless ^Q, 

and P*(q^) ^ 0. It is an overdetermined system of linear conditions on P*(x). It is 
easy to see that these polynomials are unique within a scalar factor provided they 
exist. We shall specify the normalization in section 4. We call these polynomials 
shifted Macdonald polynomials. 

It is well known that the number qi~ x t is the (q, t)-analog of the content of the 
a square s = Together with G. Olshanski we have conjectured (unpublished) 

the following analog of the formula (1.2). By 

a'(s)=j-l, l'(s)=i-l 

denote the arm-colength and the leg-colength of the square s = Then 

(1.4) P;(x;q,t) = Y,MQ,t) Y[t l T s) (x T(s) - , 

where the sum is over all reverse tableau on \x with entries in {1, 2, ... } and ifj T {q, t) 
is the same (q, i)- weight of a tableau which enters the combinatorial formula for 
ordinary Macdonald polynomials (see [M],§VI.7) 

(1.5) P^{x;q,t) = ^2^ T (q,t) [Jx TW • 

T s€/i 

The coefficients ipr(q,t) are rational functions of q and t. 

However, only in the present paper we obtain a proof of this formula. 

First theorems about the polynomials P^(x) were obtained by F. Knop and 
S. Sahi [KS,K,S]. In particular, they identified the highest degree term of the inho- 
mogeneous polynomial P*(x). Other fundamental properties (such as integrality) 



Therefore the natural 
nomial 

of degree 

which is symmetric in 
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were also established. Their approach was based on (/-difference equations for poly- 
nomials P*(x). 

The combinatorial formula (1.4) implies both the highest degree term identifi- 
cation [K,S] and the extra vanishing property (1.3) proved in [K]. It is easy to see 
that all conditions from the definition of P^(x) are obvious in (1.4) except for the 
symmetry in Xit~ l . We were not able to find any direct proof of this symmetry and 
shall give a quite indirect proof. 

We shall use an approach based on a q-integral representation for the polynomial 
P*(x). It is independent of the difference equations approach and generalizes the 
coherence property of quantum immanants (see [00], §10 and also [Okl], §5.1). 

Our main technical tool is a (/-integral representation for Macdonald polynomials 
(see Theorem I below) . Since Macdonald polynomials satisfy a (/-difference equation 
it is natural to expect that 



can be written as a multiple (/-integral where Xi occur as limits of integration. 
Indeed, P^(xi, . . . , x n ) can be written as such a multiple integral of the Macdonald 
polynomial 



in a smaller set of variables with respect to a multivariate analog of the symmet- 
ric beta measure. In the simplest case n = 2 this integral reduces to a particular 
case of the (/-analog of the beta integral studied in [AA,AV]. In the Schur function 
case this integral can be evaluated explicitly and gives the determinant ratio for- 
mula. The integral representation allows to characterize Pp{x) (and also P*(x)) as 
eigenf unctions of commuting integral operators, see remark 4.7. 

There are important (/-integral formulas involving Macdonald polynomials due 
to Kadell (see [M], example VI. 9. 3 and also [Ka]) as well as integral representation 
of Macdonald polynomials via ordinary contour integrals (see [AOS] and references 
therein) . 

The crucial property of our integral is that the domain of integration is of the 
form 



This allows to obtain a (/-integral representation for the polynomial P* just by a 
minor modification of the integral, see section 4. In other words, the orthogonal 
polynomials P^ and the Newton interpolation polynomials P* have essentially the 
same integral representation. 

In a sense the relationship between P M and P* is even closer than between s M and 
s*. One explanation for this is that the finite difference calculus (which is suitable 
to s* ) unifies with the ordinary calculus (which is suitable to s M ) in the (/-difference 
calculus. 

The lower degree terms of the inhomogeneous polynomial P* carry some im- 
portant additional information and make some properties of P* look even more 
natural than the corresponding properties of the ordinary Macdonald polynomials. 




P M (yi,...,y n -i) 
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For example, the duality for shifted Macdonald polynomials has a clear combina- 
torial interpretation (see section 6) which gives a new interpretation of the duality 
for Macdonald polynomials. 

In section 7 we consider shifted Jack polynomials. This degeneration was also 
considered by F. Knop and S. Sahi in [KS]. Even if one is interested in Jack poly- 
nomials only it proves to be easier to work with general Macdonald polynomials 
and then let q — > 1 in very final formulas. 

It would be very interesting to find a (g, £)-analog of the formula (1.1). 

I am grateful to I. Cherednik, S. Sahi and especially to G. Olshanski for many 
helpful discussions. I would like to thank R. Askey, A. N. Kirillov, K. Mimachi, 
M. Noumi, M. Wakayma for their remarks on the preliminary version of this paper 
( |q-alg 9605013| ). In particular, K. Mimachi gave me a copy of the preprint [MN]. 

Since the completion of this paper binomial type formulas for P* and also for 
ordinary Macdonald polynomials were obtained (see [Ok3] and also [003]). In 
particular, they provide new and, perhaps, more natural ways to prove the q- 
integral representations obtained here. Among other application let us mention the 
papers [KOO] and [Ok5]. 

The analogs of P* for the root system of type BC n were considered in [Ok4]. 
Those polynomials have essentially all properties of P* except for the difference 
equations. 

§2 Notations 

All (/-shifted factorials in this paper will be with the same base g, for example 
(a)oo = (1 - a)(l - qa)(l - q 2 a) .... 
This product converges if \q\ < 1. Put 



a 



The numbers q and t = q e will be the two parameters of the Macdonald polynomials 
Pfj,. With this notation 

/ n, (°)oO 

(ta) oo 

We shall need also another (/-shifted power 

(a) r = (a - l)(a - q) . . . (a - q r ~ l ), r = 0, 1, 2, . . . , 

which is a (/-analog of the falling factorial power of a. 
Recall the definition of the (/-integral 

f(y)d q y= / f(y)d q y- / f(y)d q y. 



qy — i j \yj ^qy ; j \yj ^qy ? 
b Jo Jo 
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where 



We have 



/ f{y)d q y = a{l-q)Y,fW)q i . 



(2.D j° v r i,V = 



a r+1 



where 



r 1 1 " V r 

[r\ = 



1-q 

is the (/-analog of the number r. Recall also the following (/-analog of the beta 
function integral [GR, 1.1 1.7] 

(2.2) / y a - 1 {qy) b _ 1 d q y = B q (a,b), 

Jo 

where > 0, b ^ 0, -1, -2, . . . , 

Btnb)- h^EM 
B ^ b) ~ T q (a + b) ' 

and, finally, 

T q (a) = (l-q) 1 - a (q) a . 1 . 

Given two vectors 

x = (xi,...,x n ), y=(yi,...,y n -i), 
where n = 2, 3, . . . , write 

y -< x 

if 

yi E [xi,x i+ i], i = l,...,n-l. 

Denote by 

V(x) = Yl(xi-Xj) 

i<j 

the Vandermonde determinant in variables x±, . . . , x n . Put 

V e {x) = V{x)\[{qx i /x j )e- 1 . 

We will integrate symmetric polynomials in yi, . . . ,y n -i over the domain y -< x 
with respect to the following measure 

(2.3) d(3{y I x) = V(y) JJ^gyt/x^e-i d q y , 

i,3 
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which is a multivariate analog of the beta measure in (2.2). Here 

d q y = d q y! . ..d q y n -i ■ 

With the Macdonald notation 

(t%iyj)oo 



n(*.K,, ( ) = n ( )c 



we have 



(2.4) dp{y | x) = V(y) U{l/x, ty; q, q/t) d q y , 

By A denote the algebra of symmetric functions with coefficients in rational 
functions in q and t. By A(n) denote the algebra on symmetric functions in n 
variables with the same coefficients. 

§3 ^-INTEGRAL REPRESENTATION OF MACDONALD POLYNOMIALS. 

Given a partition \i with the number of part £(n) < n put 

(3.1) Ciji, n) = H B q (fn + (n- i)6, 9) . 

In this section we will prove the following theorem 
Theorem I. Suppose < n, then 

(3-2) — / PM dP(y | x) = Cfa n)P,(x) . 

V \ X ) Jy<x 

On the one hand this formula expresses the Macdonald polynomial P^ix) in 
terms of the Macdonald polynomial P^,{y) in the smaller set of variables. By sta- 
bility of Macdonald polynomials it suffices to know _P M in variables. Therefore 

(3.2) gives a (/-integral representation of P^. This formula is a generalization of the 
determinant ratio formula for Schur functions (see below). 

On the other hand, since _P M form a basis in the space of symmetric polynomials, 
(3.2) tells how to integrate symmetric polynomials over the domain y -< x with 
respect to the measure (2.3). 

In the limit q — > 1 the formula (3.2) becomes the integral representation for Jack 
polynomials found by G. Olshanski [01]. 

Both sides of (3.2) are analytic functions of q and t in the polydisc 

\q\ < 1, \t\ < 1 . 

More precisely, if 

\q\, \t\ < S < 1 , 
then we have to assume, for example, that 

5 1/2 < \ Xi \ < <T 1/2 , i = l,...,n 
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and that 

%i 7^ X j 1 ^7^ 3 

in order to avoid zero factors. 

Therefore it suffices to prove the equality (3.2) for 

(3.3) t = q\ = 2,3,.... 

Indeed, an analytic function, which vanishes on (3.3) should vanish on all lines 

q = const 7^ 
by the 1-dimensional uniqueness theorem. 
Lemma 3.1. Suppose f(yi) is a polynomial. Then 

1 



(3.4) 



xi - x 2 JX2 



Xl 



f(yi)d q yi 



is a symmetric polynomial in x\ and x 2 . 
Proof. Follows from (2.1). □ 

The following computation is from [00], §10. 
Lemma 3.2. Suppose f(y) is a symmetric polynomial. Then 

(3-5) f f(y)V(y)d q y 

is a skew- symmetric polynomial in x. 

Proof. Since f(y)V(y) is skew-symmetric, it is a linear combination of the following 

/ y? 

det 



determinants 



for some numbers v\ > v<i > 



Integrating we get up to a constant factor 



det 



x 



— X 



Which equals 



(3.6) 



det 



x 



X 



n-1 



i/„_l+l 



(x\ 



X 



X 



rX f 



-1+1 
-1+1 

-1+1 



1\ 
1 

1/ 



To see this subtract in (3.6) the second line from the first one, then the third line 
from the second one and so on. Clearly, (3.6) is skew-symmetric in x. □ 

Remark 3.3. In the Schur functions case q = t the computation from the previous 
lemma gives in fact an explicit computation of the integral in the LHS of (3.2) 
which shows that (3.2) is a generalization of the determinant ratio formula for 
Schur functions. 

Denote by I(fi,n) the LHS of (3.2). 
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Proposition 3.4. Suppose f(y) is symmetric polynomial. Then 
(3.7) _L_j^j (y)dmx) 

is a symmetric polynomials in x of degree deg /. 

Proof. Multiply both the integrand and denominator in (3.7) by 

(n-l)(0-l) 



n 



so that they become polynomials. The integrand becomes a skew-symmetric poly- 
nomial in y with coefficients in symmetric polynomials in x, therefore by lemma 3.2 
the integral is a skew-symmetric polynomial in x. Denote this polynomial by J. 

By lemma 3.1 the polynomial J is divisible by (x\ — x 2 ). 

Since the integrand vanishes at the points 

yi =xi/q, ...,x 1 /q e ~ 1 , 

as well as at the points 

Vi = x 2 /q,...,x 2 /q ~ 1 , 
we can replace the integration 

/ d q y x 

by integration 

I d q y u s = l,...,0-l, 

J x 2 

or by 

d q yi, s = l,...,0-l. 



I x 2 /q s 

Therefore J is divisible also by 

x\ - q s x 2 , s = 1, . . . , 6 - 1, 

and 

q s xi — x 2 , s = 1, . . . , 9 — 1. 
Since J is skew-symmetric in x it is divisible by 

6-1 

VWUllixi-qSxj). 

i=£j s=l 

Therefore (3.7) is a symmetric polynomial in x. It is clear that its degree equals 
deg / + (n - l)(n - 2)/2 + (n - 1) - n(n - l)/2 = deg / , 
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where the summands come from f(y), V(y), integration, and V (x) respectively. 
This concludes the proof. □ 

Denote by n) the LHS of (3.2). We want to apply the Macdonald operator 
to 7(/i, n). Here 

\Tq,Xif \ \ x li ■ ■ ■ i x ii ■ ■ ■ i x n) = f\ x li ■ ■ ■ i Q x ii • • • ? x n) ■ 

The operator (3.8) will be also denoted by D(q, t) and by D x (q, t) when it should be 
stressed that it acts on variables x. The Macdonald polynomials are eigenfunctions 
of this operator [M] 

(3.9) dp^ = (xy iV 

We shall need the three following general lemmas about the operator D. For 
these lemmas we do not need any special assumptions about q and t. 

Lemma 3.5. For all q and t we have the following commutation relation 

(3.10) 0(i/ 9 ,i/ t )_ = kU£-— D{lMq ). 

Proof. Direct computation. □ 
Lemma 3.6. Put 

t j X i 1 j X j 



then 



(3.11) D 1/x (q,t)=t n - L D x (l/q,l/t). 

Proof. Direct computation. □ 

By definition (3.8) the operator D acts on symmetric polynomials in n variables. 
This action indeed depends on n; in other words it is not compatible with the 
restriction homomorphisms 

(3.12) A(n) -> A(n - 1) . 

In the next lemma we shall deal with two finite sets of variables xi, . . . , x n and 
yi, . . . ,y m and we suppose for simplicity that n > m. Denote by D x and D y the 
operators D in variables x and y respectively. Put 

Il n)m = U(x,y;q,t) . 
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Lemma 3.7. 

(3.14) D x U n , m = (t n ~ m D y + [n- m] t ) n n , m , 
where [n - m] t = (1 - t n ~ m ) / (1 - t) . 

Proof. As explained in [M, VI. 4] (this is clear from (3.9)) the following modification 
of the operator (3.8) 

n 

(3.15) E = t~ n D-J2 t ~ l 

l 

is compatible with homomorphisms (3.12) and therefore defines an operator 

A, 

which is self-adjoint. By [M, VI.2.13] 

E x U(x, y; q, t) = E y U(x, y; q, t) . 



Therefore 



t n yE x + l j n n)7 



i 

m 



= (t n - m D y + [n-m] t )U n , m . □ 

In particular, 

(3.16) AriW-i = (tD y + 1) n n , n _! . 

Proposition 3.8. 

(3.17) D(l/q, l/t)I(^ n) = q'^'^ /(/*, n) . 

Proof. Recall that we consider the case 6> = 2, 3, Put 

I = [ PMdp(y\x). 
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By (3.10) we have to prove that 

( ? /f) n-1 £>(l/9, */«)/« 
By (3.11) it is equivalent to 



D 1/x (q,q/t)I = ^q-^f-^Io 



We have 



(3.18) T qA/xi I = ... / .../ T q , 1/xi P^y)d(3{y\x). 

Jx 2 Jxi/q Jx i + 1 Jx n 

Since the integrand in (3.18) vanishes if 

Vj,Vj-i = Xj/q, j^i, 
we can rewrite (3.18) as follows 



T q ,i/ x Jo = / T qA/x .Pp(y) df3(y \ x) 

Jy-<x/q 



Therefore 



Di/ x (q,q/t)I = / D l/x (q,q/t)P fl (y)df3(y\x) 

Jy^x/q 



(3.19) 
Now by (3.16) 

(3.20) D 1/X (q, q/t)U(l/x, ty; q, q/t) = (jD y {q, q/t) + l) U{l/x, ty; q; q/t) . 
By (3.20) and (2.4), the integral (3.18) equals 



(3.21) 



y~<x/q 



PMV(y) (±D v (q,q/t) + l)n(l/x,ty;q;q/t) d q y . 



Consider one summand in (3.21) 



(3.22) 



* Jy-<x/q 



P»{y)v(y) 



m/t - yj 



n 



T qtVi U(l/x,ty;q;q/t) 



d q y, 



where % = 1, . . . , n — 1. Replace qyi by a new variable. To simplify notation denote 
this new variable by y^. Then (3.22) becomes 



1 pxi/q rxi px 

(3.23) -/ .../ .../ 

1 Jx 2 /q Jx i + 1 Jx n 



Xn-l/q 

/q 



Vi/t ~ Vj 



n 



Tl/q, yi PM 



d/3(y\x) 
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Since the beta measure vanishes if 



(3.24) 



Vj = x j/Qi x j+i/Qi 



the integral (3.23) equals 



(3.25) 



y-<x 



n 



Vi/t - Vj 
Vi - Vj 



T l/q,Vi P M 



d/3(y | x) 



By the same vanishing of the beta measure for (3.24) 
(3.26) 



PMViyMl/x, ty; q; q/t) d q y = / P^y)V(y)U(l/x, ty; q; q/t) d q y . 

y-<x/q Jy-<x 



Therefore the integral (3.21) equals 



(3.27) 



h+ \ I [D y (l/q,l/t)P^y)\ d(3{y\x) 

t J y-<x 



It is well known (and follows, for example, from the formula for the scalar product) 
that 



(3.28) 



P^x;q,t) = P fl (x;l/q,l/t) 



By (3.9) and (3.28) 



'n-\ 



D y (l/q,l/t)PM = P,(y) 



Hence (3.27) equals 



as desired. □ 

Corollary 3.9. /(//, n) equals the Macdonald polynomial P^(x) up to a scalar fac- 
tor. 

We calculate this factor in the next proposition. It will conclude the proof of the 
theorem. 

Consider the highest monomial in of n) with respect to the lexicographic 
ordering of monomials 
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Proposition 3.10. The highest monomial in 7(/x,n) equals 

(3.29) C(/*,n)n*?- 

Proof. Multiply both the integrand and denominator in I(fj,,n) by 

J^(_l)(^-l)(™-^-(™-W-l)/2 x (™-l)0_ 

Then the highest monomial in the denominator equals 

j-j-^e-iXn-i) _ 

i 

Calculate the highest term of the integrand. We have to give Xi and yi the same 
priority because we obtain Xi integrating j/j. Therefore the highest term of the 
integrand is 



JJ x (»-*-i)(«-i)^+(»-*-i) + (»-0(^i) (a;i _ qyi) _ {xi _ q e-i yi) = 

i 



Now calculate the highest term of the integral. The terms which come from the 
lower limits are negligible, therefore the highest term of the integral equals 

Hxt 1 - 1 ^ r yr in - i)e -\xi - m) ■ ■ ■ fa ~ f-'vi) d q y, 

Jo 

= IT xf- l - 1)(e - 1)+ ^ +(n ^ )e+e - 1 f z w+(»-0«-i(i - qz )...{\- qO-'z) d q z 

Jo 

= l[B q (^ + (n-z)e,0)x^ +in -^ 29 - 1) 

=cM]jxr {n - i){29 - i \ 

i 

where we used change of variables yi = zxi, beta function integral (2.2) and defini- 
tion (3.1). Therefore the highest term of the ratio equals 

C(^n)l[x^. □ 

i 

Example 3.11. The integral representation (3.2) gives one more way to calculate 
the special value (see [M], VI. 6. 17) 

(3.30) P^M,* 2 ,...,*"" 1 ). 

What is special about this value is that for 

9=1,2,... 

only one summand, corresponding to the point 



does not vanish in (3.2). 
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§4 (/-Integral representation for shifted Macdonald polynomials 

Denote by A*(n) the algebra of polynomials f(x\, . . . ,x n ) which are symmetric 
in new variables 

We shall call such polynomials shifted symmetric. The algebra A* (n) is filtered by 
degree of polynomials. Denote by A* the projective limit of filtered algebras A*(n) 
with respect to the homomorphisms 

A*(n) -> A*(n-1), 
/(xi, . . . , X n ) l-> /(xi, . . . , x„_i, 1) . 

The algebra A* can be naturally identified with the algebra of those polynomials 
in Macdonald commuting difference operators which are stable (that is compatible 
with homomorphisms (3.12)). An example of such operator is the operator E from 
(3-15). 

The graded algebra corresponding to the filtered algebra A* can be naturally 
identified with the algebra of symmetric functions in variables x\. The algebra A* 
is generated, for example, by the following analogs of the power sums 

(4.1) pj(s) = l)*** 1 "*), fc=l,2,.... 

i 

Given a partition \i put 

n(fj) = £(i ~ 1)^ = Yl^'M ~ l)/2 • 

i 3 

Recall that for each square s = G £t the numbers 

a(s) = fjii-j, a'(s)=j-l, 

l(s) = h'a — i, l'(s) =i — l, 

are called arm-length, arm-colength, leg-length, and leg-colength respectively. By 
definition, put 

H(n) = t - 2n ^q n ^ J] ^)+i t l(s) _ ^ 

sen 

This number will play the same role as the hook-length product played in [00]. 

Suppose £(fi) < n. By P* denote the element of A*(n) which satisfies the two 
following conditions 

(4.2) degP; = |//|, 

(4.3) P;{q x ) = H{n) 5x,» , |A| < \»\,e(\) < n , 

where q x = (q Ai , . . . , q "). Here H(fi) is just normalization constant and is intro- 
duced for convenience. Note that the condition (4.3) is weaker than the condition 
(1.3) in the introduction. Here we have a square system of linear equations on P*. 
We shall prove (1.3) below in (4.11). 

It is clear that if P* exists then it is unique. The existence of P* will follow from 
an explicit formula for it. The existence of this polynomial was proved by different 
methods in [K,S]. 

In the same way as in [00], Ex. 3.5, we have 
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Proposition 4.1. The sequence 



{P; e A*(n 



n>l(fi) 



defines an element of A* 
Proof. The polynomial 



p;{x u ... 



Xn-l, 1) 



satisfies all conditions for the shifted Macdonald polynomial in (n — 1) variables 
provided n > £(fi). □ 

Proposition 4.2. Suppose \x n > and pirf 



(4.4) P; (xi, . . . , i„) = g 1 ^ 1 J] (x 4 t 1 "* - t 1 "™) P;_ (xi/g, • • • , x„/g) . 



Proof. We have to verify that the RHS of (4.4) satisfies all conditions for P*. It is 
clear, that its degree equals \fi\. Evaluate it at q^. We have 



Finally, it is easy to see that it vanishes at all points q , where |A| < and 
£(X) <n. □ 

In this section we shall obtain a g-integral formula for polynomials P* , which is 
a minor modification of the formula (3.2). 
We shall consider the following integration 



li = (//i-l,...,/^-l). 



Then 



5 lA» \ t -n(n-l) "Q ( ? A i£ n-i _ ^ = _ 




Introduce new variables 



X> ^ ^ 



y* = Vi t 



n—l—i 



and put 



V*(x) 
V* e (x) 
d(3*(y\x) 



V(x*), 

V e (x*), 
d(1(y*\x*). 



We have 
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Theorem II. Suppose £{n) < n then 

(4-5) [ P;{y)df3*{y\x) = t\»\C{^n)P;{x). 

v \ x ) Jy<*x 

Together with (4.4) the theorem gives a formula for polynomials P*. In the proof 
we shall assume that 

(4.6) = 1,2,.... 

Indeed, the LHS of (4.5) is analytic in the polydisc \q\, \t\ < 1 and being a polynomial 
vanishing at certain points provided (4.6) must be such a polynomial for the other 
values of q and t as well. 

We need the two following elementary lemmas. 



Lemma 4.3. For any A 



(4.7) M/t),-! = a9 _, e{ e-i)/2 

(q/A)o-i 

Lemma 4.4. For any partition fx and 9 = 1,2,... 

(4.8) V* 9 (q^)^0. 

Proof of the theorem. Denote by I*(p,,n) the integral in the LHS of (4.5). First 
show that it is an element of A*(n). In variables x* , y* it can be rewritten as 

(4.9) ...J^ 
The polynomial 

p;{v) 

is symmetric in y*. Because of the vanishing of the beta measure we can replace 
integration 



by integration 



Thus, by proposition 3.4 it is a symmetric polynomial in x*, that is an element of 

A*(n). ' 

The main point is the verification of the vanishing condition. Let us evaluate 
n) at 

x = q x , 
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where A is a partition. Then the summands which enter the (/-integral correspond 
to points 

y i = q 1 % Ai>7i>Ai+i, i = l,...,n-l. 

(Recall that q < 1 and therefore q^ 1 is in fact the upper limit of integration for 
Ui and q Xi+l is the lower one. This explains also the minus sign below in (4.10).) 
Note that 7 is a partition 

7l > 72 > ' • • > 7n-l 

and I7I < |A|. 

Now suppose that |A| < \n\ and A 7^ \i. Then always 

7^ 

By definition, P*(y) vanishes at all such points! Since by (4.8) the denominator 
does not vanish, I*(/j,,n) vanishes. 

The only calculation left is evaluation of the integral at 

x = q», 

which means 

x* = q? . 

In particular x n = x* n = 1. This evaluation is elementary but quite messy. To the 
end of the proof the indices i and j will range from 1 to n — 1. Only one summand 
in the integral does not vanish. This summand equals the product of the following 
factors: 

Hip) 

from the value of P*{y) at this point, 

t-^-^-^nix* - x*) 

i<j 

from the Vandermonde determinant, 

from the other factors in the beta density, and finally 

(4.10) (-l)™- 1 ^ - qT-H-^-VJlx* 

from the q-Lebesgue measure d q y* . 
The denominator equals 

i<j i^j i 
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By (4.7) 

T T (i'i/^^ / X j)e-1 _ -0(0-l)( n -l)(n-2)/2 

Again, by (4.7) 

(-!)<(! -g)(9A) g -i(^IA) e -i _ 

(x* - l){qx*)e-i{q/x*) e -i ~ (x*) e 

which equals 

(x*) e q ^B q (^ + (n-z)O,O). 
Therefore the result equals 

C(fi,n)H(ri 

times the following power of q 

%| + 9 2 n(n - l)/2 - 0(0 - l)(n - 1) 

- 0(0 - l)(n - l)(n - 2)/2 - 0(n - l)(n - 2)/2 - 0(n - 1) 

which equals 0\(i\. □ 
Write 

/iCA 

if Aj > jLtj for all i, and 

/i C A 

if jU C A and \x ^ A . In the sequel we shall deal with some sums like 



where the only thing that matters is which P v can enter the sum, no matter with 
which specific constant factors c v . Constant factor like c v will always denote such 
unspecific factors. 

The following proposition is a strengthening of a result by S. Sahi and F. Knop.In 
particular, it gives the highest degree term of P*, which was found in [K,S] by 
different methods. 

Proposition 4.5. 

(4.11) P*( Xl ,x 2 ,...,x n ) = P^(x 1 ,x 2 t-\...,x n t 1 - n ) + 

C„P„(x 1 ,X2t~ 1 , . . . ,X n t 1_n ) , 

vCfJ, 
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where c u are some constants 1 which depend on fx and n. 
Proof. Induct on n and Suppose fi n > 0. By (4.4) we have 

P;{x u ...,x n ) = I ( J2(-t~ n ) k e n _ k (. . . , Xl t x ~\ . . . ) ) P M *_ ( Xl /q, . . . , xjq) . 

\fc=o / 

By inductive assumption (4.11) is true for fi~ . It is well known that the product 

&r P^ 

is a linear combination of such Macdonald polynomials P v that 

rj/iy 

is a vertical r-strip. Therefore (4.11) is true also for \i. 
Suppose [i n = 0. By inductive assumption 

P*(y u . . . , y n _i) = P^yu ■ ■ ■ , Vn-i t 2 ~ n ) + c Mvi' • • • > Vn-i t 2 ~ n ) ■ 
Therefore 

p;( yi , . . . , yn _o = ^ 2 -^ip M ( yi , . . . , + £ c^(^, . . . , y^) , 

for some constants c'^. Integrating as in (3.2) and (4.5) we obtain 

Pfi ( x li • • • > x n) = t^ 1 n ' ) ' M 'P At (x 1 , . . . , X n ) + C^P;,^, . . . , X n ) , 

which is equivalent to (4.11). □ 

Using (4.5) we can reprove one more result from [K], which generalizes the cor- 
responding result for the shifted Schur functions [00], Th. 3.1. 

Proposition (Vanishing property) 4.6. 

(4.12) P^(q X ) = 0, unless ll C A 

Proof. Induct on n and If (i n > apply proposition 4.2. 

If fj, n = then apply the very same argument which proved vanishing in the 
proof of the theorem. □ 

Remark 4.7. The formulas (3.2) and (4.5) are certain (/-integral operators 

A(n- 1) -> A(n) 

1 Explicit formulas for these coefficients are given by a particular case of the binomial theorem 
for Macdonald polynomials, see the recent paper [Ok3] 
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and 

A*(n-l)->A*(n). 

Using projection (3.12) (and its analog for A*) in the inverse direction we can 
characterize Macdonald polynomials and shifted Macdonald polynomials as eigen- 
functions (with distinct eigenvalues) of some integral operators (in the same way 
as for Schur functions [00], §10). 

In fact, there are countably many commuting integral operators which corre- 
spond to iterated integration (3.2) and (4.5) 

A(n) — > A(AT), n<N, 

and projection back 

A(N) -> A(n) . 

§5 Combinatorial formula for shifted Macdonald polynomials. 

In this section we shall establish branching rule for shifted Macdonald polynomi- 
als, or (what is the same) a combinatorial formula for P* in terms of semistandard 
tableaux on [i. 

Fist, we need some qualitative results about the branching for P*. 
Proposition 5.1. Let d be a variable, then 

(5.1) P*(dxi,...,dx n ) = y^ j c„(d)P*(x 1 ,...,x n ), 

where c u (d) are some polynomials in d which depend also on [i and n. 
Proof. It is clear that an expansion 

P*(dx 1 ,...,dx n ) = y~]cy(d)PZ(xi, . . . ,x n ) 

exists. We have to show that 

c v {d) = 0, unless v C /j, . 

By (4.11) the bases 

(5.2) {P;(x u ...,x n )} 
and 

(5.3) {P^.^Xit 1 -*,...)} 

are mutually triangular with respect to the partial ordering by inclusion of diagrams. 
Since dilatation by d in diagonal in the basis (5.3) it is triangular in the basis 
(5.2). □ 

Put 

\l! = . . . , fln-t) . 

We have 
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Proposition 5.2. 

(5.4) P*(xi,...,X n _i,d) = ^(^P^Xi^.^Xn-i), 

where c v {d) are some polynomials in d which depend also on [i and n. 

Proof. Again we have to show that certain summands cannot occur. 
If d = 1 then by stability of P* 



P*(x 1 ,...,x n - 1 ,l) = | 



P*(xi,...,x n _i), //„ = 

0, /Un>0. 



For general d introduce new variables & 

Xi = d^, z = l,...,n-l, 



and use (5.1). □ 
Put 

Recall that 

means that 



>= (fl 2 , ■ ■ ■ , Vn) ■ 
V -< fi 

A*l > ^1 > A*2 > • • • > f n -l > A*n , 



which is equivalent to 

Vci/C/i'. 

Proposition 5.3. 

(5.5) P*(d, X 2 , • • • , X n ) = ^ U,M Ki X 2: ■ ■ ■ , X n ) , 

where f^ jU (d) are some polynomials in d. 

Note that because of stability of shifted Macdonald polynomials the polynomial 
fn,u(d) does not depend on n. 

Proof. By the shifted symmetry 

P*{d, x 2 , ■ ■ ■ , x„) = P*{x 2 /t, . . . , x n /t, t n_1 rf) . 

Hence by (5.4) 

p;(d, X2 ,...,x n )=j2 c l/ (t n - i d)p:( X2 /t, . . . , x n /t) . 

i/C/x' 
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By (5.1) 

P*(x 2 /t,...,X n /t) = ^ Cr,(l/t)P*(xi, ...,X n ). 

rj<Zi/ 

Therefore only summands with 

can occur in (5.5). 

Now assume that there is a summand in (5.5) with 

We can choose v minimal, that is in such a way that P* with rj C v do not enter 
the sum (5.5). Then for all sufficiently large d 

P^d, !/!,...,!/„_!) ^0 

for 

and all other summands in (5.5) vanish by the vanishing property (4.12). Clearly, 
this contradicts (4.12) and makes our assumption impossible. □ 

Now recall the corresponding result for Macdonald polynomials 

(5.6) P^d , X 2 , • • • ? %n 

where V , m, z/ are cer tain nonzero rational functions of q and t which can be found in 
[M], section VI. 7. Introduce the following notation 

where the numbers a'(s) and l'(s) for a square s£|i were defined in section 4. The 
number q a ( s h~ l ^ is the (/-analog of the content of the square s. If 

i/ = 0, n = (r) , 

then 

Therefore the number (d)^/^ is a generalization of the factorial power (d) r . The 
main result of this section is the following 

Theorem III. 

(5.7) P*(d,x 2 ,...,x n ) = ^W,^ _H (d)^ /v P*{x2,...,x n ), 

It is clear that iterating this formula we obtain the semistandard tableaux sum 
formula (1.4) for polynomials P*. We shall use induction on n. In fact we shall 
need only the following corollary of this theorem 
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Corollary 5.4. For all r = 1, . . . , n. 

(5.8) P; On, . . . , x r , ,...,q^) = cx^...x^ + ..., 

where c is a nonzero factor and dots stand for lower monomials in lexicographic 
order. 

Proof. Given a partition 
set 

{t) V = (Vi,---,Vn-l) ■ 

Then 

C =t-^^ ( ^i(n^ +1 ),)^ +i v). □ 

Proof of the theorem. Induction on n. The case n = 1 is clear. 
Suppose n > 1. We shall find 

l/VH 

distinct zeros of the polynomial f^^d). Fix some i and show that 

(5.9) /„,„(<*) = 0, d=q^-H 1 -\... 1 q v H 1 - i . 

We shall prove (5.9) by induction on ^z/, in other words we shall deduce (5.9) from 
the assumption that 

(5.10) U,v(d) = 0, d = q^-H 1 -\...,q^t 1 -* 
for all 7] such that 

Suppose d is as (5.9). We have 

(5.11) P;(d, x 2 ,...,x i ,q"%..., q^) = ]T f^{d) P*(x 2 , . . . , a*, g"*, . . . , q v ^) . 

By the vanishing property (4.12) only summands satisfying 

W77 C {i) v 

are nonzero. On the other hand, if 
then in particular 

?7i < i/t 
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and by our assumption (5.10) the corresponding summand vanishes. Therefore only 
summands with 

enter the sum. By (5.8) each summand has the following form 
(5.12) cf,, n {d)(xf...xf^ + ...), 

where c is a nonzero factor and dots stand for lower monomials in lexicographic 
order. 

On the other hand, by shifted symmetry 
(5.14) P;(d, x 2 ,..., Xi ,q»%..., «f - 1 ) = P;(x 2 /t, . . . , Xi/t, f-'d, q»%..., q^) . 
By the vanishing property (4.12) this should vanish if d is as in (5.9) and 

Xi/t = q K , 

for all sufficiently large integers 

A 2 > • • • > A, . 

Therefore for such d the polynomial (5.11) should be identically zero. By virtue of 
(5.12) it is impossible unless 

UM = 

for all r) such that = Mj,. This proves (5.9). 
Since 

the polynomial /^(cf) equals 

up to a factor. This factor is clear from (4.10) and (5.6). □ 

Example 5.5. The shifted analogs of the elementary symmetric functions are 

(5.14) P* lk) (x)= £ t k -^^]J(x is -t s - k ). 

ii<---<ik s 

Remark 5.6. The shifted symmetry of P* results in some complicated identities 
for coefficients VVi" which are not clear from the explicit formula for these coeffi- 
cients. 
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§6 Duality 

The duality we shall discuss in this section relates shifted Macdonald polynomials 
with parameters q and t to shifted Macdonald polynomials with parameters 1/t and 
1/q. Denote by 

A? 

the algebra A* constructed in section 4. Denote by 
(6.1) pt(x;t) = J2^ 1 - i \x^-l) 

i 

the power-sum generators of A£. Consider the following isomorphism uj* t 



A? AJ/,, 

(6-2) pt(t) " ^y^k P*(Vg)- 

Note a slight difference with the Macdonald duality automorphism. The isomor- 
phism (6.2) has the following clear combinatorial interpretation which generalizes 
the corresponding result for shifted Schur function [00], Th. 4.1. 

Proposition 6.1. For all /G A ( * 

(6-3) Kt(/)](*- A ) = /(? A ')- 

Proof. It suffices to check (6.3) on generators (6.1). For any partition A we have 

< 6 - 4 > AE' w (^ - u = ir^iy-H*-* 5 - 1). 

« j 

which is example VI. 5. 1(a) in [M]. To see (6.4) one can use the same trick as in 
[00], Th. 4.1, and sum 

first along rows and then along columns. It is clear that (6.4) means 

' ' pl(t-\l/q)=pl(q X '-t). 



1 - 1/t 

Now replace q by q k and t by t fc in (6.4) to obtain 

' Pt(t- x ;l/q)=pt(q x ';t), 



as desired. □ 
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Corollary 6.2. 

(6.7) < t (p;M)) = c p;,(i/MAz), 

where c is a constant which depends on fx. 
Proof. Follows from (6.3) and definition of P*. □ 
Recall the following notation of Macdonald 

bx(q,t) = [[— (a)+w - 



sex 



In particular, 



by{t,q)-[[— {B)Ka)+1 
sex * 



Theorem IV. 

(6.8) <tOP;M)) = (-t)^(t, q )p;,(i/t,i/ q ). 

Proof. It is clear that in order to calculate the constant in (6.7) we have to evaluate 
the equality (6.7) at t~^ . By (6.3) and definition of P* we have 

= H(/i;q,t) 
° H(»'; 1/t, 1/q) 

t -2n(») q n(»') ]l seM (? a(s) + 1 t' (s) " 1) 
~~ qZn(p,')t-n(p,) n seM (?~ a(s) t~' (s)_1 - 1) 

= (-t)\ X \b x ,(t,q), 
^l(s)=n(ji), = 7i(//') • □ 



because 



Example 6.3. Since the automorphism (6.2) preserves the degree we can look at 
the corresponding isomorphism of the graded algebras 

A -> A, 

which differs from Macdonald automorphism 
by factor 

(-t) deg , 

where deg is the degree of a polynomial. Since the highest term of P* is the 
Macdonald polynomial P M and 

P li (q,t) = P li (l/q,l/t), 

the theorem (6.8) gives a new proof of the duality theorem of Macdonald together 
with computation of the constant factor. 
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§7 Degeneration 

In this section we consider shifted Jack polynomials. These polynomials were 
considered by F. Knop and S. Sahi in [KS]. These polynomials are indexed by 
partitions fx, depend on a parameter 9 and equal, by definition, to 

(7.1) P;{x ; 9) = hm(g - P*(tf- q, q e ) . 

In this section let us change notations as follows. The dependence on q will be de- 
noted explicitly in all prelimit expressions like in the RHS of (7.1). The expressions 
without q will denote the corresponding degenerations. For example, we write 

(7.2) {z;q) r = (z-l)(z-q)...(z-q r - 1 ). 
This is the shifted Macdonald polynomial in one variable 

P*(z;q,t) = {z;q) r . 
The corresponding shifted Jack polynomial equals 

P*(z;9) = {z) r , 

where 

(7.3) (z) r = z{z-l)...{z-r+l). 
In the same way write 

<*>/*/-= II (z-a'(s) + 9l'(s)), 

sen/ v 

this is a generalization of (7.3). 

The easiest way to see that (7.1) is a polynomial of degree \\x\ is to look at the 
combinatorial formula for P*(q,t). The branching rule (5.7) has the following limit 

(7.4) p;( Xl , 3^2? • • • , X n 

where ^^^{9) are the branching coefficients for ordinary Jack polynomials, which 
are rational functions in 9 and can be found in [St] and [M], section VI. 10. Note 
that 

6=1/ a, 

where a is the traditional parameter for Jack polynomials. 

Because of the symmetry and vanishing properties of the shifted Macdonald 
polynomials we have 



(7.5) Pu( x 5 0) i s symmetric in variables Xi — 9 i , 

(7.6) P*(X ; 9) = unless fj, C A , 

(7.7) P;(iu;9) = H(a(s) + 9l(s) + l). 

sen 
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The product in (7.7) is a ^-analog of the hook length product; denote it by H{ji ; 6). 

Before discussing the limit of integral formula (4.5) for P*(x ; q, t) consider some 
elementary properties of finite sums. We have 



(7-8) £ (y) 



(Xl + l) r+ l - (X 2 ) r +1 

r — 



r + 1 

y=x 2 



provided x\ — x-i is a nonnegative integer. Using (7.8) one can evaluate sums 
(7-9) £ /(y) 



y=x 2 



for arbitrary polynomials /(y). If x\ — xi is not a nonnegative integer then define 
(7.9) using (7.8). In particular, we have 

Xi X2 — 1 

E=- E 

y=x 2 y=X!+l 

Now suppose that / is a polynomial and 

lim , 1 r f( q v) = J(y) 

for some s and /. Then 

(7-10) lim 1 +1 f /(y) rf,y = £ /(y) , 

which follows directly from definitions if xi — x 2 is an integer. 
Now we calculate the limit of (4.5). Recall that [GR,1.10] 

\imT q (z)=T(z), 

<?— 1 

where q tends to 1 from below. Therefore 

(cf)e-i T(z + 9-l) 

«™ (i - (z^- 1 r(z) 

In the sequel we assume that 

(9 = 1,2,.... 

In this case 
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Recall that we denote by V* e {x ; q) the denominator in (4.5) 

V* {x ; q) = H(q 9 ^ Xi - q 6 ^ Xj ) \{{q e ^ +1 Xi / Xj ) 6 -i . 

i<j i^j 

We have 

(7ii) ii m — ( gX; ^ = (_ 1 \n(n-i)/2y*e () 

where 

(7.12) V* e (a:) = - ^ + 0{j - i)) X\{ Xl - x 3 + 9{j -i) + 6- l) e -i . 

Denote by 

(3* (y, x ■ q, q e ) = IJ^"- 1 "'^ - g^"" 1 "^) H(q e ^ +1 - 9 yi / Xj )e-i 
the density of the beta measure in (4.5). In the same way we have 
(7-14) lim P ^ qV, S: ( Q,q 2 9W9 = (-l)^-^ n -^ 2 P*(y, X ;e) , 

v ' q-*\ (1 — q\n(n-l)0+(n-l)(n-2) /2 v > f vi " ' > ' 

where, by definition, 

(7.15) 0* (y, x ; 0) = IJ( W " Vi + °(J ~ 0) II<W - x i + " z ) " • 
Using (7.10), (7.11), and (7.14) we obtain the following degeneration of (4.5) 

(7.16) y^— p ^y ; e ) /%> x ^) = c ^ n ) p > ; *) > 

where 

E = E- E • 

and 

C(//,n) = n% + («"^ <?)• 

For = 1 we obtain the coherence property of the shifted symmetric functions 
[00], §10. 
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